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ABSTRACT 

A  technique  for  graphical  assembly  of  large  jigsaw  puzzles  by  computer 
vision  is  presented.  The  solution  uses  the  puzzle  piece  shape  information 
only.  The  pieces  are  photographed,  digitized,  their  boundary  information  is 
extracted  and  a  local  matching  algorithm  is  applied  to  compute  the  matching 
scores  between  every  two  sides  of  different  puzzle  pieces.  Then  a  global 
matching  algorithm,  based  on  combinatorial  optimization  techniques,  is 
applied  to  compute  the  correct  assembly  of  the  puzzle  pieces.  The  frame  of 
the  puzzle  is  solved  by  the  "traveling  salesman"  algorithm  and  the  the  interior 
of  the  puzzle  is  solved  by  a  branch  and  bound  algorithm.  The  method  was 
used  successfully  to  assemble  104  piece  puzzles,  which  have  many  pieces  with 
almost  similar  boundaries.  The  technique  was  extended  to  assemble  two 
intermixed  104  piece  puzzles. 


1.   Introduction. 

1.1.  In  this  paper  we  describe  a  technique  for  solving  jigsaw  puzzles  by 
computer  vision  and  report  the  experimental  results  of  applying  our  technique  to 
104-piece  jigsaw  puzzles.  In  our  approach  each  piece  of  the  puzzle  is  photographed 
and  digitized  and  its  boundary  is  calculated.  Using  only  this  boundary  data,  our 
algorithm  calculates  a  global  matching  between  these  puzzle  pieces  and  consequently 
produces  an  assembly  of  the  puzzle  (as  shown  in  Fig.l  and  Fig. 5).  Similar 
techniques  are  applied  to  a  successful  solution  of  an  intermixed  double  puzzle.  Two 
104-piece  puzzles  were  proccessed  together  with  no  a-priori  indication  that  this  is 
the  case.  The  computer  program  is  able  to  determine  that  it  is  dealing  with  two 
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separate  puzzles  and  then  assembles  each  of  them  successfully. 

1.2.  The  jigsaw  puzzle  assembly  problem  is  regarded  as  a  strenous  test  for  2-D 
curve  matching  techniques  and  was  studied  in  [F-G](1964),  where  a  9  piece  puzzle 
was  assemled.  using  matching  techniques  based  on  the  Freeman  chain  code  and 
some  heuristics,  and  in  [R-B](1982),  where  a  4  piece  puzzle  was  assembled  using 
curve  matching  based  on  the  so  called  "boundary-centered  polar  encoding" 
technique. 

In  both  these  cases  as  in  our  approach  the  puzzle  is  turned  upside  down,  so  the 
only  information  used  is  the  shape  of  the  pieces  (in  [F-G]  it  is  called  an  "apictorial" 
puzzle).  In  [R-B]  the  puzzle  pieces  were  traced  on  a  data  tablet.  Both  [F-G]  and  [R- 
B]  require  a  high  degree  of  discrimination  between  the  boundary  curves  of  different 
puzzle  pieces  to  enable  successful  puzzle  assembly. 

1.3.  Our  approach  uses  as  its  basis  ths  Schwartz-Sharir  curve  matching 
algorithm  (see  [S-S]).  We  use  this  method  to  assemble  "apictorial"  jigsaw  puzzles  of 
more  than  100  pieces,  many  of  which  are  similarly  shaped  (see  Fig. 3  and  Fig. 4). 
We  are  using  commercial  children's  puzzles  (e.g.  Fig.l  shows  the  other  side  of  the 
"Mickey  Mouse  &  Donald  Duck"  puzzle  by  Jaymar)  the  pieces  of  which  were 
turned  upside  down  and  photographed  separately  by  a  black  and  white  camera.  This 
photographic  procedure  introduces  additional  noise  which  does  not  exist  when  the 
input  data  is  obtained  artificially. 

1.4.  Because  of  these  three  factors  -  a  relatively  large  number  of  puzzle  pieces, 
many  strong  similarities  between  boundaries  of  different  pieces  and  the  additional 
level  of  noise,  it  is  difficult  to  obtain  a  complete  puzzle  assembly  just  from  the  basic 
matching  algorithm  and  straightforward  heuristics.  Hence  the  results  of  the  "local" 
Schwartz-Sharir  matching  algorithm  are  taken  as  input  to  a  second  "global"  matching 
which  uses  these  results  and  assembles  the  puzzle  using  further  combinatorial 
optimization  techniques. 

1.5.  Since  the  basic  local  matching  algorithm  is  described  in  detail  in  [S-S],  we 
give  only  a  short  overview  of  it  in  section  4  and  this  paper  is  devoted  mainly  to  the 
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description  of  the  global  algorithm  which  is  given  in  section  5  and  the  double  puzzle 
solution  which  is  described  in  section  6.  However,  it  should  be  noticed  that  the 
good  performance  of  this  local  algorithm  is  an  essential  component  in  our  puzzle 
assembly.  In  fact  from  our  experiments  it  is  clear  that  the  puzzles  with  the  simpler 
sizes  and  shapes  used  in  previous  works  ([F-G],[R-B])  could  be  easily  assembled 
using  only  the  local  matching  algorithm,  without  any  global  considerations.  Indeed, 
in  a  previous  experiment  we  have  managed  to  assemble  a  simpler  15  piece  puzzle 
using  only  local  matching. 

2.   Deflnition  of  the  problem. 

2.1.  We  consider  two-dimensional  rectangular  jigsaw  puzzles  of  about  100 
pieces  (see  for  example  Fig.l).  Every  interior  puzzle  piece  has  four  neighbours,  and 
every  frame  piece,  except  the  four  corner  pieces,  has  three  neighbouring  pieces. 
The  puzzle  is  arranged  in  a  grid.  In  most  of  the  cases  observed,  four  adjacent  pieces 
meet  at  a  common  corner.  However,  this  condition  does  not  hold  everywhere 
(notice  at  least  four  "irregular"  junctions  in  Fig.l)  and  it  is  not  a  prerequisite  for 
successful  implementation  of  the  proposed  algorithm.  It  will  also  become  clear  from 
the  following  discussion  that  the  puzzle  need  not  even  be  rectangular,  although  in 
such  a  case  we  need  some  other  criteria  to  distinguish  the  frame  pieces  of  the 
puzzle. 

2.2.  In  what  follows  we  describe  the  process  of  the  puzzle  assembly  algorithm 
step  by  step  refering  specifically  to  the  examples  of  Fig.l  and  Fig. 5,  which  are 
13x8=104  piece  jigsaw  puzzles  of  overall  size  18"xl3".  The  intermixed  puzzle 
experiment  has  used  the  same  two  puzzles.  We  used  puzzles  of  the  same  size,  so 
there  will  be  no  way  to  distinguish  between  the  pieces  of  the  different  puzzles  by 
some  "external"  parameter,  such  as  size. 


3.   Preprocessing. 

3.1.  The  puzzle  pieces  are  photographed  one  by  one  by  a  black  and  white  RCA 
2000  camera  and  the  pictures  are  digitized  and  threshholded  to  get  a  binary  image 
for  each  puzzle  piece. 

3.2.  The  boundary  of  each  piece  is  extracted  from  the  binary  image  and  a 
polygonal  approximation  of  it  is  obtained.  This  polygonal  approximation  smoothes 
the  boundary  and  eliminates  some  of  the  noise  from  it.  It  is  needed  in  order  to 
apply  successfully  the  [S-S]  matching  algorithm.  All  subsequent  processing  is  done 
on  these  polygonal  approximations  of  the  original  boundaries. 

3.3.  The  boundary  curve  of  each  piece  is  divided  into  four  subcurves 
corresponding  to  the  four  sides  of  the  puzzle  piece  and  these  curves  are  later  used  in 
the  matching  procedure.  Two  different  methods  have  been  applied  in  order  to  find 
the  four  corners  of  each  piece. 

A  first  such  method,  which  has  also  been  used  in  [K-S-S-S]  to  find  the  so  called 
"breakpoints"  in  a  boundary  of  a  composite  scene,  first  finds  all  the  points  on  the 
boundary  of  the  piece  at  which  the  tangent  has  a  sharp  change  in  direction  and, 
hence,  the  second  derivative  reaches  a  peak.  Since  there  may  be  more  than  four 
points  on  the  boundary  with  this  property,  a  heuristic  procedure,  based  on  the 
standard  shape  of  the  pieces,  is  applied  to  extract  the  four  corners. 

The  second  corner-finding  technique  exploits  the  fact  that  at  corners  the 
boundary  curve  closely  approximates  a  right  angle.  More  specifically  a  small  portion 
of  a  right  angle  is  matched  against  the  boundary  curve  using  the  [S-S]  matching 
algorithm  and  this  gives  us  the  four  most  likely  corner  points.  Application  of  this 
method  is  in  a  preliminary  stage  and  some  refinements  are  presently  being 
developed. 

3.4.  Pieces  having  an  (almost)  straight  section  between  adjacent  corners  are 
identified  as  "perimeter"  or  "frame"  pieces. 


4.   Local  matching  procedure. 

The  next  paragraphs  give  a  short  overview  of  the  2-D  curve  matching  algorithm 
of  [S-S].  For  a  more  detailed  exposition  see  [S-S]. 

4.1.  Take  two  curves  C  and  C  in  the  plane  and  assume  that  C  is  a  translated 
and  rotated  subcurve  of  C .  Both  C  and  C  are  assumed  to  have  been  smoothed  (i.e. 
they  are  polygonal  approximations  of  the  original  curves)  and  parametrized  by  arc 
length  s.  The  matching  we  seek  calls  for  determination  of  the  offset  sq  and  the 
Euclidean  transformation  E  for  which  the  curves  EC{s)  and  C'{s  +  sq)  are  closest  to 
one  another  in  the  l}  norm.  To  be  more  specific,  we  represent  each  of  the  curves 
C,  C  by  a  sequence  of  evenly  spaced  points  on  it,  and  let  these  sequences  be 
(u,)^=l  and  (v,)JLj  respectively.  Assume  first  that  both  curves  have  the  same 
starting  point  (i.e.  Sq=Q  and,  hence,  m^  n).  Matching  thus  amounts  to  finding  a 
Euclidean  motion  E  of  the  plane  which  will  minimize  the  /  distance  between  the 
sequences  (Euy)"=i  and  (Vj)"=i: 


n 


-  '  A  =  min    ^\EMj-yjY 

To  simplify  the  calculation,  first  translate  C  so  that 

Next  write  E  as  Eu  =  R^n  +  a,  /?9  denoting  a  counterclockwise  rotation  by  0. 
In  such  case,  as  it  is  shown  in  [S-S],  the  best  match  is  obtained  when 

1     " 

and  9  is  the  negation  of  the  polar  angle  of  ^  ujVj,  where  the  vectors  u^,  v^  are 
regarded  as  complex  numbers  uj,  vy.  The  least-square  distance  for  this  best  match 
is  given  by 

A  =  i   KP  -  7li  v,P  +   ±   |u,P  -  2|i  ujvjl  (♦) 


If  the   curves   do  not  have  the   same   starting  point,   we  have  to  match  the 
sequence  (u,)j'=i  to  each  of  the  contiguous  subsequences  (Vj+j)"=i  of  the  sequence 
(Vj)7=i,  forrf  =  0.  .  .  .  ,m-n. 
For  each  such  d  (*)  thus  becomes 

We  seek  the  minimum  of  the  values  A(^),  d  =  Q,  .  .  .  .m-n,  which  can  be  found  in 
time  0(m  logm),  using  the  fast  Fourier  transform  algorithm  for  computing  the 

n  _ 

convolutions  2  "j^j+d- 

Remark:  Though  in  most  junction  poiiitu  the  matching  curves  have  the  same 
starting  point,  we  deliberately  ignore  this  iilfcimatioil  in  order  to  make  the 
performance  of  the  algorithm  more  general. 

4.2.  The  local  matching  procedure  desciibed  abcve  is  applied  to  every  pair  of 
'quarter  boundary'  subcurves  of  the  puzzle  pieces  (sec:  3.3).  Hence,  if  we  have  an 
A^-piece  puzzle,  we  get  an  4N*4N  symmetric  matrix  of  the  best  matching  scores 
between  every  two  such  sides. 

It  will  be  clear  from  the  description  of  the  global  algorithm  (see  Section  5)  that 
we  do  not  have  to  compute  all  the  entries  of  this  matrix,  since  the  frame  pieces  are 
treated  separately,  nevertheless  we  still  require  0{N^)  matchings. 

5.   The  puzzle  assembly  algorithm. 

5.1.  The  puzzle  assembly  algorithm  consists  of  two  major  substeps.  The  frame 
of  the  puzzle  is  assembled  first,  and  then  it  is  used  as  a  starting  point  for  assembly 
of  the  entire  puzzle. 

Frame  assembly 

5.2.  Assembly  of  the  puzzle  frame  may  be  viewed  as  the  assembly  of  a  one- 
dimensional  puzzle. 


Once  the  frame  pieces  have  been  recognized  (see  3.4)  their  straight  hne  side  is 
known.  Suppose  that  we  name  the  sides  of  the  piece  in  the  order  of  their  appearance 
counterclockwise  -  bottom,  right,  top  and  left  side,  where  the  straight  line  is  the 
bottom  side(  see  Fig. 3).  For  the  four  corner  pieces  we  will  have  two  bottom  sides 
and  one  right  and  left  side. 

It  is  easy  to  see  that  a  correct  matching  along  the  outer  frame  may  occur  only 
between  a  right  side  and  a  left  side. 

5.3.  Define  a  matrix  M(i,j),  ij  =  \,...,K,  where  K  is  the  number  of  the  frame 
pieces,  by  agreeing  that  M{i,i)  should  equal  the  inter-curve  L  distance  obtained  by 
best  matching  the  right  side  of  piece  /  to  the  left  side  of  piece  ;.  Any  choice  of 
exactly  K  eniriss,  one  from  each  row  and  one  from  each  column,  defines  a  K- 
permutation  P  cf  ;he  frjme  piew.*  so  that  P(/)  =  y,,  where  y,-  is  the  entry  chosen 
from  the  i-th  row. 

Our  task  is  to  find  K  entries  in  M,  jne  in  each  row  and  one  in  each  column,  in  such 
a  way  that  the  sum  of  these  K  entries  is  minimal  and  the  AT-permutation  P  is  a  cycle. 

Suppose  that  these  entries  have  j,aen  determined  to  be  j\,...,  Jk,  where  y,  is  the 
entry  in  the  i-th  row.  Then  we  can  assemble  the  outer  frame  by  simply  putting  piece 
ji  next  to  the  right  of  cierc  i. 

The  combinatorial  problem  that  we  have  just  stated  is  well  known  in  the 
literature  as  the  traveling  salesman  problem  (see,  for  example  [C]  or  [L-L-RK-S]), 
and  is  usually  formulated  as  follows: 

Given  K  cities  and  the  distances  between  each  pair  of  them,  find  the  shortest  closed 
path  which  passes  through  every  city  exactly  once. 

In  our  case  we  must  solve  the  so  called  asymmetric  problem,  which  means  that  the 
distance  from  city  /  to  city  ;  is  not  necessarily  equal  to  the  distance  from  city  ;  to 
city  /. 

5.4.  The  traveling  salesman  problem  is  known  to  be  A'P-complete  (see  [G-J]), 
but  for  small  AT's  (less  than  100)  there  are  efficient  algorithms  that  solve  it  (see,  for 
example    [L-L-RK-S]). (    Note    that    in    our    example    A:  =  2*(13  +  6)  =  38    .)    These 
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algorithms  may  be  divided  into  two  main  categories  ; 

(a)  time-efficient  algorithms  which  use  heuristic  methods  and  therefore  only  assure 
discovery  of  the  optimal  solution  with  high  probability  (see  for  example 
[R].[C]). 

(b)  algorithms  which  always  find  the  optimal  solution,  but  in  some  cases  may  be 
very  time  consuming  (see  [B-M]). 

We  actually  use  the  algorithm  of  type  (a)  which  was  published  in  [F].  This  type 
of  algorithm  is  preferable  for  our  purpose,  because  it  guaranties  fast  performance. 
Since  we  plan  to  implement  our  solution  by  a  robot  (see  [B-W]),  possible  errors  in 
the  proposed  solution  will  be  corrected  by  the  following  interactive  procedure  : 

(1)  apply  the  traveling  salesman  algorithm  to  th2  problem  defined  by  matrix  M  and 
feed  the  solution  to  the  robot. 

(2)  check  the  proposed  solution  by  the  robot  and  return  to  the  computer 
information  about  correct,  incorrect  and  (possibly)  undecidable  matches 
between  suggested  neighbouring  pieces. 

(3)  if  all  the  matches  are  correct  -  go  to  the  next  step.  If  some  of  the  matches  are 
incorrect  -  change  matrix  M  according  to  the  information  obtained  from  (2), 
assigning  zero  entries  to  the  correct  matches  and  "almost  infinity"  to  the 
impossible  matches  and  go  to  (1). 

After  a  very  small  number  of  such  iterations  the  correct  solution  should 
emerge.  In  our  experiments,  we  have  obtained  the  correct  frame  assembly  after  the 
first  application  of  the  traveling  salesman  algorithm  on  the  original  matrix.  This 
happens  with  high  probability,  because  the  matrix  M  is  not  random,  but  strongly 
favorable  to  its  optimal  solution  by  virtue  of  the  good  performance  of  the  local 
matching  algorithm.  We  therefore  expect  in  most  cases  to  get  the  optimal  solution 
by  using  the  algorithm  of  [F]  without  any  feedback.  As  mentioned  this  has  been  the 
case  in  our  experiments. 

5.5.  If  we  take  the  traveling  salesman  problem  and  drop  the  condition  that  P 
should    be    a    cycle,    we    get    another    well   known    problem,    which    is    called   the 


assignment  or  bipartite  matching  problem  (see  [C],[L]).  There  is  an  efficient  solution 
to  this  problem  with  a  polynomial,  (i.e.  0(K^  ))  time  complexity.  Many  type  (b) 
solutions  to  the  traveling  salesman  are  based  on  this  algorithm  (see  [L-L-RK-S])  and 
in  our  case  we  believe  that  in  most  applications  even  this  less  constraining 
assignment  algorithm  will  find  the  correct  assembly  of  the  outer  frame. 

In  section  6  we  describe  the  use  of  the  assignment  algorithm  in  order  to 
assemble  the  frames  of  a  number  of  intermixed  different  puzzles  simultaneously. 

Remark:  As  is  seen  from  the  above  description  of  the  algorithm,  the 
rectangular  form  of  the  puzzle  is  inessential  to  correct  arrangement  of  the  frame, 
since  we  deliberately  ignore  the  fact  that  the  right  and  left  sides  of  matching  pieces 
begin  from  the  same  bottom  starting  point,  and  the  bottom  straight  line  is  used  only 
to  recognize  the  frame  pieces  and  their  outer  side.  All  that  we  actually  need  in  case 
of  non-rectangular  puzzles  is  an  alternative  method  of  obtaining  the  same 
information.  For  example,  the  puzzle  could  be  circular  or  the  outer  frame  side  could 
be  simply  marked  as  such  in  some  manner. 

Arrangement  of  the  puzzle  interior 

5.6.  After  complete  arrangement  of  the  outer  frame  (see  Fig. 2),  we  go  on  to 
the  arrangement  of  the  interior  pieces.  Basically,  this  is  done  by  considering  the 
four  interior  corners  of  the  frame  (see  Fig. 2)  and  using  the  fact  that  an  interior 
piece,  which  fits  into  one  of  the  four  corners  of  the  frame  has  to  match  two  sides 
with  two  previously  known  frame  pieces.  This  is  advantageous  because  the 
matching  curve  is  approximately  twice  as  long  compared  to  the  previous  case. 

5.7.  A  simple  approach  to  solve  the  interior  is  to  use  a  greedy  algorithm.  Using 
such  an  approach  we  could  start  with  the  piece  which  has  the  best  matching  score 
with  one  of  the  corners.  After  such  a  piece  is  discovered  and  located,  it  will  create 
two  other  corners  with  the  same  property.  Then,  we  could  look  for  another  piece 
having  the  best  possible  match  with  one  of  the  remaining  corners,  and  proceed 
iteratively  in  this  way.  Of  course,  at  some  stage  we  will  get  places  in  which  a  piece 
must  match  along  three  of  its  sides;  this  strengthens  our  scoring  procedure  even 
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more.  Finally,  the  last  piece  will  have  to  match  along  four  of  its  sides. 

5.8.  However,  in  our  situation,  in  which  the  boundaries  of  the  pieces  are  quite 
similar,  even  when  two  sides  are  being  matched  (see  Fig.4)  a  greedy  algorithm  is 
unlikely  to  succeed.  Hence  some  backtracking  or  branch  and  bound  algorithm  is 
necessary.  For  this  we  use  the  following  approach: 

Corners  are  processed  sequentially,  begining  with  the  lower  left  side  of  the  puzzle 
interior  and  advancing  within  each  row  to  the  right.  (This  procedure  is  less  general 
than  picking  up  the  best  corners  wherever  they  may  be,  but  it  is  easier  to  program 
and  the  loss  of  information  is  non-essential). 

For  the  first  corner,  all  the  puzzle  pieces,  which  are  not  frame  pieces  are  located  at 
this  corner  in  all  possible  rotations  and  their  loca!  matching  score  is  computed.  The 
results  are  sorted  and  a  prescribed  number  of  best  solutions  (denoted  as  KBEST)  is 
passed  to  the  next  stage. 

At  the  second  corner  the  same  procedure  is  repeated  for  everyone  of  the  KBEST 
partial  solutions  which  passed  the  previous  stage,  and  only  KBEST  overall  best 
solutions  are  passed  to  the  next  stage. 

The  algorithm  then  proceeds  iteratively.  At  the  last  corner  in  each  row  we  have 
three  sides  to  match.  In  the  last  row  we  have  three  matching  sides  for  every  piece 
and  four  matching  sides  for  the  last  piece. 

Of  course,  the  number  KBEST  need  not  be  the  same  for  all  corners  but  it  is 
more  appropriate  to  vary  it  from  stage  to  stage.  A  more  sophisticated  approach 
would  be  to  make  a  dynamic  decision  as  to  which  solutions  should  pass  to  the  next 
stage  by  assigning  an  upper  bound  (as  a  function  of  the  stage)  for  the  overall 
matching  of  a  partial  solution  and  to  pass  along  only  those  solutions  that  do  not 
exceed  this  bound. 

In  our  experiments  (Fig.l  and  Fig. 5)  we  used  KBEST  =  200,  which  was  kept  the 
same  at  every  stage.  Actually,  the  correct  solution  in  the  puzzle  of  Fig.l,  for 
example,  always  lay  among  the  10  best  solutions. 
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5.9.  In  the  next  few  paragraphs  we  describe  the  way  in  which  simultaneous 
matching  of  two  sides  of  a  piece  is  accomplished. 

The  most  accurate  way  is  to  arrange  the  frame  (or  a  given  partial  solution), 
take  a  picture  of  the  relevant  corner  and  then  try  to  match  this  curve  with  the 
boundary  curves  of  the  remaining  pieces.  However  this  method  is  too  tedious  for  a 
computerized  assembly. 

A  reasonable  approximation  to  such  a  method  is  to  use  available  information 
about  the  relative  matching  angles  and  displacements  of  the  pieces  and  to  calculate 
the  curve  formed  by  joining  two  sides  at  a  corner.  This  method  has  a  disadvantage 
of  accumulating  angular  errors  and  may  result  in  slightly  distorted  linkup  between 
the  curves  meeting  at  a  corner  junction. 

To  H'-'oid  this  accumulation  of  ei-ors  while  not  losing  the  information  that  the 
two  curves  being  matched  at  a  corner  must  belong  to  the  same  piece,  we  compute 
*  the  sum  of  the  scores  of  both  sides  when  they  are  taken  independently,  and  we  add 
to  this  sum  a  penalty  score  which  is  proportional  to  the  excessive  shift  of  the  side 
curves,  if  they  do  not  meet.  This  procedure  enables  us  to  precompute  the  matrix  of 
all  possible  best  matches  and  their  relative  displacements  for  every  pair  of  curves 
(see  4.2),  eliminating  the  need  for  a  local  matching  procedure  to  be  re-invoked 
within  the  main  loop  of  the  global  algorithm. 

The  puzzle  of  Fig.l  was  solved  correctly  even  without  using  this  penalty  score 
on  wrong  displacements,  i.e.  by  considering  both  sides  independently;  however,  to 
solve  correctly  the  puzzle  of  Fig. 5  we  had  to  use  the  penalty  score.  We  assume  that 
the  difference  lies  in  the  accuracy  of  the  data  aquisition  step. 

6.   Solution  of  an  Intermixed  double  puzzle. 

6.1.  To  test  our  methods  in  a  still  more  strenuous  manner,  two  104-piece 
puzzles  (Fig.l  and  Fig. 5)  were  intermixed  and  treated  as  one  208-piece  puzzle. 

Preprocessing  and  local  matching  were  done  in  the  same  way  as  described  in 
Sections  3  and  4. 


-  12- 

However,  the  puzzle  assembly  algorithm  was  changed  to  reflect  the  new  fact 
characterising  this  sort  of  intermixed  puzzle  problem,  since  this  time  we  had  to  drop 
the  (wrong)  assumption  that  one  connected  puzzle  has  to  be  assembled,  i.e.  that  the 
frame  permutation  should  consist  of  a  single  cycle. 

6.2.  The  frame  pieces  were  recognized  and  separated  as  before,  but  the  frame 
assembly  was  not  done  using  the  traveling  salesman  algorithm,  which  always 
produces  one  cycle  of  a  permutation  P  (see  5.3  and  5.4),  but  by  the  assignment 
algorithm  (see  5.5).  The  number  of  the  cycles  in  the  permutation  obtained  should 
then  be  equal  to  the  number  of  different  puzzles  being  processed  simultaneously  and 
the  size  of  the  cycles  correspond  to  the  size  of  the  puzzle  frames. 

Once  the  frames  of  the  different  puzzles  are  obtained  we  can  arrange  their 
interior  by  the  methods  described  in  5.6-5.8.  Note  however  that  since  v/e  have  no 
way  to  distinguish  between  the  pieces  of  the  different  puzzles  (except  the  fiames) 
the  assemblies  of  the  different  interiors  must  proceed  in  parallel. 

6.3.  In  our  experiment  we  obtained  two  (correct)  frame  cycles  using  the 
assignment  algorithm  and  then  arranged  the  interior  of  one  of  them  by  the  methods 
of  5.6-5.8,  when  all  the  remaining  puzzle  pieces  (of  both  puzzles)  were  considered 
as  candidates  for  the  vacant  places.  Once  one  of  the  puzzles  was  assembled 
correctly,  we  were  left  with  the  problem  of  assembling  only  one  104-piece  jigsaw 
puzzle,  as  done  before.  This  approach  was  chosen,  merely  because  it  required  only 
minimal  changes  in  the  existing  single  puzzle  assembly  program  ;  of  course,  more 
sophisticated  technical  details  could  be  developed. 

Remark:  In  the  experiment  described  above  we  used  a  local  matching  score 
which  takes  into  account  the  relative  displacement  of  the  puzzle  pieces  both  for  the 
solution  of  the  interior  and  the  frame  (see  5.9). 
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7.    Future  experiments  and  research. 

7.1.  It  is  quite  obviou.s  that  the  same  methods  can  be  applied  to  bigger  puzzles. 
In  fact,  our  double  puzzle  experiment  can  be  viewed  as  a  "more  difficult"  problem 
than  an  ordinary  208-piece  puzzle  assembly. 

7.2.  For  puzzles  which  do  not  have  a  grid  like  form  other  heuristics  need  to  be 
developed.  This  more  general  case  suggests  the  following  fundamental 

Problem:  given  two  curves,  find  the  longest  matching  subcurve  which  appears  in 
both  curves. 

A  good  algorithm  for  solving  this  problem  would  make  it  possible  to  apply  our 
global  algorithm  to  arbiirary  puzzles,  without  additional  heuristics. 
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Fig.l  :  An  assembled  104-piece  jigsnw  puzzle 


Fie-.  2  :  An  assembled  frame  of  a  lOA-piece  puzzle 


Fig. 3  :  Two  different  frame  pieces 


Fig. 4  :  Three  different  puzzle- interior  pieces 


Fig.  5  :  An  assembled  lOA-piece  jigsaw  puzzle 
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